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Abstract — We  formulate  concepts  that  characterize  net¬ 
work  properties  in  the  presence  of  continuously  occurring 
faults,  and  we  present  CPV,  a  path-vector  routing  pro¬ 
tocol  that  locally  contains  continuously  occurring  faults 
and  locally  stabilizes.  Local  containment  enables  CPV  to 
protect  distant  nodes  from  being  affected  by  faults.  Local 
stabilization  enables  CPV  to  stabilize  the  network  within 
time  depending  on  the  perturbation  size  after  faults  stop 
occurring.  In  CPV,  the  distance  to  which  the  state  of  a 
node  propagates  is  proportional  to  the  time  the  state  re¬ 
mains  valid.  These  properties  are  achieved  by  reacting  to 
new  fault  only  after  first  containing  the  response  to  the 
previous  fault.  In  addition  to  analytically  proving  these 
properties,  we  evaluate  CPV  by  simulating  Internet-type 
networks  with  up  to  75  autonomous  systems;  we  observe 
that  CPV  reduces  the  number  of  fault-affected  nodes  by  a 
factor  of  71  and  the  network  convergence  time  by  a  factor 
of  9.2  when  compared  with  BGP. 

Keywords — containment  of  continuously  occurring  faults, 
local  stabilization,  path-vector  routing,  BGP,  Internet 

1  Introduction 

A  well-known  ideal  in  networking  and  distributed  com¬ 
puting  is  the  ability  to  withstand  failure  or  compromise 
of  one  or  more  regions  in  a  network  without  impacting  a 
large  part  of  the  network  [6,  19,  2].  To  this  end,  formal 
models  and  mechanisms  of  fault  containment  have  been 
proposed  and/or  used  in  practice  [7,  3,  2,  17].  These 
models  and  mechanisms  have  largely  focused  on  cases 
where  faults  stop  occurring  after  certain  moment  in  time, 
faults  occur  with  low  frequency,  or  faults  assume  specific 
patterns  by  which  fault-occurrences  can  be  predicted. 

Nevertheless,  faults  may  keep  occurring  with  high  fre¬ 
quencies,  and  the  interval  between  faults  may  be  shorter 
than  the  time  taken  for  the  network  to  stabilize.  More¬ 
over,  complex  interactions  between  different  network  com¬ 
ponents  may  generate  unanticipated  faults,  especially 
when  networks  work  in  stressful  conditions.  Consequently, 
most  existing  mechanisms  cannot  guarantee  fault  con¬ 
tainment  in  the  presence  of  these  high-frequency  and 
unanticipated  faults.  One  area  where  this  issue  remains 
to  be  addressed  is  inter-domain  routing  via  path-vector 
protocol  BGP  in  the  Internet  [19]. 

Under  the  Code  Red/Nimda  attack,  for  instance,  mem¬ 
ory  overflow  and  BGP  session  reset  cause  routers  to  re- 
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peatedly  fail-stop  and  rejoin  at  frequencies  as  high  as 
once  every  minute  [19];  even  though  instability-suppression 
timers  (such  as  MinRouteAdvertisementlnterval  and  Mi- 
nASOriginationlnterval  [16])  and  route- flap-damping  are 
used,  faults  at  some  edge  routers  propagate  across  the 
whole  Internet  [19].  Such  unbounded  fault  propagation 
decreases  not  only  the  availability  of  networks  but  also 
their  stability  and  scalability. 

To  provide  dependable  services,  therefore,  networks 
must  be  able  to  contain  the  impact  of  high-frequency 
unanticipated  faults.  Moreover,  networks  should  con¬ 
verge  quickly  once  faults  stop  occurring,  within  time  as 
a  function  of  the  degree  of  fault  perturbation  instead  of 
the  network  size  (we  refer  to  this  property  as  local  sta¬ 
bilization).  To  this  end,  formal  models  that  characterize 
network  behaviors  and  mechanisms  that  guarantee  fault 
containment  in  the  presence  of  high-frequency  unantici¬ 
pated  faults  are  desired. 

Related  work.  The  concepts  of  fault  containment  and 
local  stabilization  have  been  discussed  in  [7],  [2],  and  [3]. 
But  the  definitions  there  are  proposed  only  for  the  cases 
where  faults  stop  or  only  state  corruption  can  occur,  thus 
they  do  not  apply  to  cases  where  faults  keep  occurring  at 
high  frequencies  and  complex  unanticipated  faults  may 
occur. 

To  locally  contain  faults  in  distance-vector  routing, 
protocol  LSRP  [2]  has  been  proposed.  Yet  LSRP  guar¬ 
antees  fault  containment  and  local  stabilization  only  for 
scenarios  where  faults  happen  at  low  frequencies  and  net¬ 
works  get  enough  time  to  stabilize  from  one  fault  before 
another  one  occurs.  Therefore,  LSRP  does  not  guaran¬ 
tee  fault  containment  when  faults  keep  occurring  at  high 
frequencies. 

To  improve  the  stability  of  BGP,  instability-suppression 
timers  and  routing-flap-damping  are  used  [16,  17].  Nev¬ 
ertheless,  they  only  deal  with  faults  of  certain  patterns 
(e.g.,  minimum  inter-fault  interval  and  fault  predictabil¬ 
ity).  Therefore,  they  do  not  guarantee  fault  containment 
in  the  presence  of  high-frequency  unanticipated  faults,  as 
experienced  in  Internet  [19]  and  observed  in  [2]  and  our 
simulation  study  in  Section  6. 

To  improve  BGP  convergence  speed  after  fault  occur¬ 
rence,  various  mechanisms  have  been  proposed  [15,  10,  4, 
20,  13].  Nevertheless,  these  mechanisms  do  not  focus  on 
fault  containment  and  thus  cannot  contain  continuously 
occurring  faults.  On  the  other  hand,  these  mechanisms 
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can  be  used  with  those  that  guarantee  fault  containment 
to  improve  network  stability  and  availability  (to  be  dis¬ 
cussed  further  in  Section  7). 

In  [7],  algorithms  are  proposed  to  contain  a  single 
state  corruption  during  the  stabilization  of  a  spanning 
tree,  but  these  algorithms  do  not  deal  with  continuously 
occurring  faults,  node  fail-stop,  and  node  join.  In  [3], 
a  broadcast  protocol  is  proposed  to  contain  observable 
variables  in  the  presence  of  state  corruptions,  but  the 
protocol  allows  for  global  propagation  of  internal  proto¬ 
col  variables  and  is  not  “stability-adaptive” . 

Contributions  of  the  paper.  To  build  the  foundation 
for  studying  and  to  precisely  characterize  system  prop¬ 
erties  in  the  presence  of  continuously  occurring  faults, 
we  formulate  the  notions  of  perturbed  node,  contami¬ 
nated  node,  perturbation  size,  contamination  range,  T- 
containment,  and  .^-stabilization.  These  concepts  are 
generally  applicable  to  networking  and  distributed  com¬ 
puting  problems. 

For  the  problem  of  path- vector  routing,  we  design  CPV, 
a  path-vector  routing  protocol  that  contains  continu¬ 
ously  occurring  faults  and  is  locally  stabilizing.  In  the 
presence  of  continuously  occurring  faults,  the  properties 
of  CPV  are  as  follows: 

•  The  only  nodes  that  are  affected  by  a  fault  are  those 
within  0(p)  distance  from  the  fault-perturbed  re¬ 
gion,  where  p  is  the  size  of  the  perturbed  region. 

•  The  distance  to  which  the  state  of  a  node  propagates 
is  proportional  to  the  sojourn  time  of  the  state  (i.e., 
the  time  for  which  the  state  remains  valid),  and  as 
a  result,  the  more  unstable  a  node  is,  the  shorter  is 
the  distance  to  which  its  state  propagates. 

Once  faults  stop  occurring  (either  indefinitely  or  for  a 
long  enough  period),  the  network  converges  to  a  legiti¬ 
mate  state  within  0(r(p'))  time,  where  p'  is  the  pertur¬ 
bation  size  of  the  faults  and  T  is  a  function  dependent 
on  the  routing  policy  used  in  the  network1. 

CPV  achieves  these  properties  via  the  following  design 
pattern.  When  a  new  fault  occurs,  before  CPV  gener¬ 
ates  a  stabilization  wave  to  correct  the  network  routes, 
it  first  contains  the  effect  of  the  stabilization  wave  re¬ 
sulting  from  the  previous  fault.  The  containment  wave 
propagates  faster  than  the  stabilization  wave  to  this  end. 
To  deal  with  the  case  where  the  containment  wave  is  it¬ 
self  propagated  in  error  (for  instance,  if  yet  another  fault 
happens  before  the  stabilization  wave  corresponding  to 
the  new  fault  is  generated),  CPV  generates  an  undo- 
containment  wave  that  propagates  even  faster  than  the 
containment  wave.  This  wave  is  self-correcting.  We  find 
that  this  pattern  is  generally  applicable  to  other  net¬ 
working  and  distributed  computing  problems. 

We  analytically  evaluate  the  properties  of  CPV  using 
the  above  concepts.  We  also  evaluate  CPV  by  simulat¬ 
ing  Internet-type  networks  with  up  to  75  autonomous 
systems  (ASes);  the  results  corroborate  our  analysis  by 

1r  is  linear  when  the  shortest-path-first  route  ranking  policy  is  used. 


showing  that,  in  the  presence  of  continuously  occurring 
faults,  CPV  reduces  the  number  of  fault-affected  nodes 
by  a  factor  of  71  and  the  network  convergence  time  by  a 
factor  of  9.2  when  compared  with  BGP. 

Organization  of  the  paper.  In  Section  2,  we  present 
the  network  model,  the  protocol  notation,  and  the  fault 
model.  We  also  briefly  recall  BGP.  To  simplify  presen¬ 
tation,  we  discuss  the  design  of  CPV  in  Section  3  before 
formulating,  in  Section  4,  the  concepts  of  fault  contain¬ 
ment  and  local  stabilization  in  the  presence  of  continu¬ 
ously  occurring  faults.  We  analyze  the  properties  of  CPV 
in  Section  5,  and  present  the  simulation  results  in  Sec¬ 
tion  6.  We  further  discuss  CPV  in  Section  7  and  make 
concluding  remarks  in  Section  8. 

2  Preliminaries 

In  this  section,  we  present  the  network  model,  the  proto¬ 
col  notation,  and  the  fault  model.  We  also  briefly  recall 
BGP. 

Network  model.  A  network  G  is  an  undirected  graph 
(' V,E,P ),  where  V  is  the  set  of  nodes  (i.e.,  BGP  speak¬ 
ers),  E  is  the  set  of  links,  and  P  is  the  function  that 
defines  the  routing  policies  of  each  node.  (In  this  paper, 
we  only  consider  routing  policy  functions  by  which  BGP 
converges.)  V  is  divided  into  several  subsets,  each  of 
which  is  an  autonomous  system  (simply  denoted  as  AS 
hereafter).  Each  node  has  a  unique  node- id,  and  all  the 
nodes  in  the  same  AS  have  the  same  AS-id.  For  a  node 
i,  the  id  of  its  AS  is  denoted  by  i.as.  For  any  two  nodes 
i  and  j,  ( i,j )  is  in  E  if  i  and  j  can  communicate  with 
each  other  directly,  or  if  i  and  j  are  in  the  same  AS. 

Message  transmission  between  nodes  is  reliable,  and 
message  passing  delay  across  a  link  is  bounded  from 
above  and  from  below  by  U  and  L  respectively.  There 
is  a  clock  at  each  node;  the  ratio  of  clock  rates  between 
any  two  neighboring  nodes  is  bounded  from  above  by  a, 
but  no  extra  constraint  on  the  absolute  values  of  clocks 
is  enforced. 

For  clarity  of  presentation,  we  only  consider  one  des¬ 
tination  d,  an  address  prefix  representing  a  set  of  nodes 
in  an  AS  d.as.  (Our  protocol  readily  applies  when  there 
are  multiple  destinations.) 

Protocol  notation.  We  write  protocols  using  a  variant 
of  the  Abstract  Protocol  notation  [8] .  At  each  node,  the 
protocol  consists  of  a  finite  set  of  variables  and  actions. 
Each  action  consists  of  three  parts:  guard,  guard  hold¬ 
time,  and  statement.  For  convenience,  we  associate  a 
unique  name  with  each  action.  Thus,  an  action  has  the 
following  form: 

{name)  ::  {guard) - - — >  ( statement ) 

The  guard  is  either  a  boolean  expression  over  the  proto¬ 
col  variables  of  the  node  or  a  message  reception  opera¬ 
tion;  h  is  the  guard  hold-time  ( h  >0);  the  statement  up- 
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dates  zero  or  more  protocol  variables  of  the  node  and/or 
sends  out  some  message(s).  If  h  =  0,  we  write  the  action 
in  the  following  form: 

{name)  ::  {guard)  — >  { statement ) 

For  an  action  whose  guard  is  a  message  reception  oper¬ 
ation,  its  guard  hold-time  must  be  0. 

For  an  action  named  a,  its  guard  hold-time  is  denoted 
by  h. a.  An  action  a  is  enabled  at  time  t  if  the  guard  of 
a  evaluates  to  true  at  t.  An  action  a  is  executed  at  time 
t  only  if  a  is  continuously  enabled  from  time  (t  —  h.a ) 
to  t.  To  execute  an  action,  its  statement  is  executed 
atomically. 

Fault  model.  A  node  or  a  link  is  up  if  it  functions 
correctly,  and  it  is  down  if  it  fail-stops.  We  consider  the 
following  network  faults:  an  up  node  or  link  can  fail-stop 
and  become  down;  a  down  node  or  link  can  become  up 
and  join  the  network;  and  the  routing  policies  of  a  node 
can  change.  The  interval  between  any  two  faults  can  be 
any  non-negative  value. 

Border  Gateway  Protocol.  BGP  is  a  path-vector 
routing  protocol  (where  a  node  maintains  the  complete 
AS-level  path  to  each  destination)  used  to  coordinate 
routing  among  ASes  in  the  Internet  [16].  In  BGP,  UP¬ 
DATE  messages  are  passed  between  nodes  to  convey 
routing  information.  BGP  UPDATES  are  route  records 
that  include  the  following  attributes  (among  others): 


nlri 

:  network  layer  reachability  information  (i.e. ,  the 

destination  address); 

nextjiop 

:  the  next  hop; 

AS-path 

:  ordered  list  of  ASes  traversed,  with  more-recently- 
visited  ASes  in  front  of  less- recently-visited  ASes; 

locaLpref 

:  local  preference; 

med 

:  multi-exit  discriminator. 

Each  route 

r  is  associated  with  a  4-tuple  rank{r),  de- 

fined  as  {r.locaLpref ,  \r,A^ath\ ,  r.nJt_hop)-  For 


the  destination  d,  all  the  routes  available  to  a  node  i 
is  ranked  in  lexical  order  by  ronfc(-),  and  the  route  with 
the  highest  rank  is  selected  as  the  route  of  i,  denoted  as 
i.aspath. 

Given  a  route  r  available  to  a  node  i,  attribute 
r.locaLpref  is  determined  by  the  route  ranking  policy  of  i. 
For  convenience,  we  call  the  ranking  policy  that  assigns 
r.locaLpref  to  a  constant  value  the  shortest-path-first  pol¬ 
icy  or  the  SPF  policy,  where  a  route  with  the  shortest 
AS-path  ranks  the  highest. 

Besides  route  ranking  policy,  BGP  uses  import  and 
export  policies.  The  import  policy  of  a  node  i  defines  the 
set  of  import  neighbors  of  i  whose  routes  are  accepted  by 
i;  the  export  policy  of  i  defines  the  set  of  export  neighbors 
of  i  to  which  i  announces  its  route. 

3  Protocol  CPV 

Our  objective  is  to  design  a  path- vector  routing  protocol 
that  satisfies  the  following  two  requirements. 


•  It  contains  continuously  occurring  faults  (whether 
or  not  anticipated)  locally  around  where  they  oc¬ 
cur,  such  that  the  distance  to  which  the  fault  effects 
propagate  is  a  function  of  their  duration. 

•  Once  faults  stop  occurring,  it  converges  from  an  ar¬ 
bitrary  state  to  a  legitimate  state  within  time  de¬ 
pending  on  the  number  of  nodes  perturbed  by  the 
faults. 

To  this  end,  we  first  describe  the  issue  of  fault  propaga¬ 
tion  in  BGP,  and  we  then  present  the  detailed  design  of 
our  protocol,  CPV. 

3.1  Fault  propagation  in  BGP 

To  see  how  faults  propagate  unboundedly  in  BGP,  we 
consider  the  simple  network  shown  in  Figure  1,  where 

Figure  1:  A  simple  line  network.  For  simplicity,  each 
node  in  the  figure  also  represents  its  AS. 

d  is  the  destination  to  which  the  other  nodes  maintain 
a  route.  Suppose  d  fail-stops  at  a  state  where  all  other 
nodes  have  learned  their  routes  to  d.  Then  e  will  with¬ 
draw  its  route  to  d  and  send  a  route- withdrawal  message 
to  /.  Now  suppose  d  rejoins  the  network  before  /  with¬ 
draws  its  route.  Then  nodes  /,  g,  h,  and  i  should,  ide¬ 
ally,  not  withdraw  their  routes  anymore.  Nevertheless, 
the  interval  between  d  fail-stopping  and  d  rejoining  and 
the  fault  history  may  well  be  such  that  BGP  instability- 
suppression  timers  as  well  as  route-flap-damping  do  not 
prevent  nodes  /,  g,  h,  and  i  from  withdrawing  their 
routes  (even  though  these  nodes  will  learn  later  the  same 
routes  again).  Thus,  in  this  fault  scenario,  the  faults 
propagate  to  all  the  nodes  in  the  network,  whereas  the 
faults  should  ideally  only  affect  e.  Due  to  the  fault  propa¬ 
gation,  the  time  taken  for  the  network  to  stabilize  after  d 
joins  is  a  function  of  the  network  size  instead  of  the  min¬ 
imum  number  of  nodes  that  should  have  been  affected 
(which  is  1  in  this  case). 

To  simplify  presentation,  we  used  a  simple  network 
and  a  simple  fault  scenario  in  the  above  discussion.  In 
practice,  networks  are  more  connected  where  multiple 
paths  exist  for  a  destination,  which  can  incur  more  net¬ 
work  instability  [20];  and  there  are  more  complex  fault 
scenarios,  with  varying  frequencies  of  fault  occurrence 
and  varieties  of  state  corruptions  [9,  11].  Therefore, 
the  negative  impact  of  fault  propagation  is  even  more 
severe  in  practice  (as  shown  in  Section  6  and  in  [19]). 
Thus,  unbounded  fault  propagation  in  networks  should 
be  avoided. 

3.2  Protocol  concepts 

One  reason  why  faults  propagate  unboundedly  in  the 
above  example  is  as  follows. 
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•  When  d  rejoins  the  network,  the  route- withdrawal  as 
the  result  of  the  fail-stop  of  d  has  already  propagated 
to  /. 

•  After  d  rejoins,  e  establishes  its  route  to  d  and 
sends  a  route-announcement  to  /;  the  route- 
announcement,  however,  lags  behind  the  route- 
withdrawal  (which  reflects  an  obsolete  state  of  the 
network)  in  the  sense  that  /  has  sent  out  the  route- 
withdrawal  to  g  before  /  receives  the  new  route- 
announcement  from  e. 

•  As  a  result,  the  route- withdrawal  keeps  propagating 
from  /  to  g,  and  then  to  h  and  i,  even  if  the  route- 
announcement  tails  it  to  reach  g,  h,  and  i. 

Design  pattern  for  continuous  containment.  To 

contain  continuously  occurring  faults,  we,  therefore,  need 
a  mechanism  that  enables  information  regarding  each 
new  network  state  to  catch  up  with  and  stop  the  prop¬ 
agation  of  information  regarding  the  preceding  state 
which  has  become  obsolete.  To  this  end,  we  design 
protocol  CPV  where  the  diffusing  computation  involved 
in  path-vector  routing  consists  of  three  diffusing  waves 
propagating  at  different  speed:  stabilization  wave  at 
the  lowest  speed,  containment  wave  at  an  intermediate 
speed,  and  undo-containment  wave  at  the  highest  speed 
(see  Figure  2).  The  three  diffusing  waves  run  in  paral- 
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Figure  2:  Parallel  diffusing  waves  in  CPV 

lei  and  coordinate  to  contain  the  propagation  of  obsolete 
information  while  stabilizing  a  network  at  the  same  time: 

•  Whenever  a  node  j  needs  to  change  its  state,  it  en¬ 
gages  a  containment  wave  cwo  before  engaging  a  new 
stabilization  wave  sw±,  so  that  cwq  stops  the  pre¬ 
vious  stabilization  wave  swq  from  propagating  the 
existing  state  of  j  (which  will  become  obsolete  once 
j  executes  swi); 

•  In  the  presence  of  continuously  occurring  faults,  an¬ 
other  fault  /  may  occur  before  j  executes  sw i ,  then 
there  are  two  cases: 

(i)  j  does  not  need  to  change  its  state  any  more 
after  /  occurs.  Then  cwq  has  to  be  stopped  so 
that  swq  can  keep  propagating  to  nodes  whose 
state  still  needs  to  be  corrected.  To  this  end,  j 
engages  an  undo- containment  wave  uwq  which 
catches  up  with  and  stops  cwq. 

(ii)  j  still  needs  to  change  its  state  after  /  occurs. 


Then  j  does  nothing  but  lets  cw o  propagate  (to 
stop  SWq). 

•  Each  stabilization  as  well  as  undo-containment  wave 
stabilizes  itself,  and  each  containment  wave  is  sta¬ 
bilized  (and  deactivated)  by  the  corresponding  sta¬ 
bilization  or  undo- containment  wave. 

We  elaborate  on  the  design  pattern  as  follows. 

A.  Containing  a  stabilization  wave:  When  a  node 
needs  to  engage  a  new  stabilization  wave  to  change  its 
state,  the  existing  state  of  the  node  becomes  obsolete 
but  may  have  propagated  to  other  nodes  via  the  pre¬ 
vious  stabilization  wave.  To  avoid  unbounded  propaga¬ 
tion  of  its  existing  state,  the  node  engages  a  containment 
wave  before  generating  the  new  stabilization  wave,  and 
the  containment  wave  will  stop  the  previous  stabiliza¬ 
tion  wave.  When  a  containment  wave  propagates  from  a 
node  i  to  its  neighbor  j,  j  further  propagates  the  contain¬ 
ment  wave  if  j  has  propagated  the  obsolete  information 
from  i.  For  instance,  in  the  example  discussed  in  Sec¬ 
tion  3.1,  when  d  rejoins,  e  will  detect  that  it  needs  to 
change  state  (i.e.,  to  establish  a  route  to  d);  therefore, 
e  initiates  a  containment  wave  toward  /,  then  /  decides 
whether  to  propagate  the  containment  wave  depending 
on  whether  /  has  propagated  the  route- withdrawal  to  g, 
and  so  on. 

To  contain  stabilization  waves  in  the  presence  of  con¬ 
tinuously  occurring  faults,  each  containment  wave  from  a 
node  i  carries  the  prediction  of  the  state  to  which  i  will 
converge  so  that  the  neighbors  of  i  are  able  to  decide 
whether  to  hold  a  stabilization  wave.2  Moreover,  to  con¬ 
tain  continuously  occurring  faults,  the  containment  and 
stabilization  waves  that  are  related  to  the  same  fault 
should  be  able  to  co-exist.  To  this  end,  containment 
waves  do  not  modify  variables  of  stabilization  waves,  and 
each  containment  wave  is  a  one-way  open  (instead  of 
a  two-way  closed)  diffusing  process  that  is  deactivated 
later  by  an  associated  stabilization  or  undo-containment 
wave. 

B.  Containment-assisting  stabilization  wave:  Sta¬ 
bilization  waves  are  initiated  or  propagated  by  nodes 
that  need  to  change  routes  after  the  occurrence  of  faults. 
To  enable  continuous  fault  containment,  stabilization 
waves  adapt  the  basic  path-vector  routing  algorithm 
with  the  following  mechanisms. 

(i)  When  a  node  selects  its  route  to  a  destination,  it 
takes  into  account  the  predicted  state  of  its  neigh¬ 
bors. 

(ii)  When  a  containment  wave  catches  up  with  the  cor¬ 
responding  stabilization  wave  at  a  node  i,  both  the 
containment  and  the  stabilization  wave  stop  at  i 
(this  is  enabled  by  not  letting  i  join  any  stabi¬ 
lization  wave).  For  instance,  in  the  example  dis¬ 
cussed  in  Section  3.1,  after  d  rejoins,  if  the  contain- 

2  Note  that  the  predicted  state  can  be  used  to  improve  packet  for- 
warding  too,  since  it  reflects  a  fresher  network  state. 
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ment  wave  initiated  at  e  reaches  /  before  /  prop¬ 
agates  the  route-withdrawal,  /  will  not  propagate 
the  route-withdrawal  anymore,  which  prevents  the 
route- withdrawal  from  propagating  further. 

On  the  other  hand,  to  guarantee  convergence  (es¬ 
pecially  in  the  presence  of  transient  loops  in  path- 
vector  routing  [14]),  a  node  that  has  already  joined 
a  containment  wave  is  free  to  join  a  stabilization 
wave  without  sacrificing  continuous  fault  contain¬ 
ment. 

(iii)  Stabilization  waves  propagate  slower  than  contain¬ 
ment  waves,  so  that  containment  waves  can  catch 
up  with  associated  stabilization  waves  which  are 
initiated  earlier. 

C.  Undoing  a  containment  wave:  In  the  presence  of 
continuously  occurring  faults,  information  that  is  obso¬ 
lete  at  some  point  in  time  may  become  valid  later  when 
other  faults  occur.  In  this  case,  the  containment  wave 
marking  that  information  as  obsolete  should  be  stopped 
and  prevented  from  propagating  further.  This  is  enabled 
by  letting  the  node  that  first  detects  this  situation  ini¬ 
tiate  an  undo-containment  wave  which  propagates  along 
the  same  path  as  the  corresponding  containment  wave. 
Since  the  undo-containment  wave  propagates  faster  than 
the  containment  wave  does,  the  undo- containment  wave 
is  able  to  catch  up  with  and  stop  the  containment  wave. 
Considering  the  example  discussed  in  Section  3.1,  for  in¬ 
stance,  when  d  fail-stops,  a  containment  wave  cwq  will 
propagate  from  e  toward  /,  and  so  on;  when  d  rejoins 
later,  e  will  initiate  another  containment  wave  cw ±  car¬ 
rying  the  route-announcement  that  is  “predicted”  to  take 
place  later;  suppose  /  has  not  withdrawn  its  route  when 
it  receives  the  predicted  route- announcement  from  e,  / 
will  detect  that  it  should  stop  cw o;  therefore,  /  initi¬ 
ates  an  undo-containment  wave  toward  g  to  stop  cw o. 
(Note  that  /  does  not  propagate  cw i,  since  /  has  not 
withdrawn  its  route.) 

For  the  parallel-diffusing-wave  based  approach  to 
work,  each  undo-containment  wave  must  self- stabilize  it¬ 
self  locally  in  the  presence  of  faults  (otherwise,  we  would 
need  another  type  of  diffusing  wave  to  contain  the  undo- 
containment  wave).  This  is  achieved  by  ensuring  that 
undo- containment  waves  use  only  those  variables  that 
are  defined  for  stabilization  and  containment  waves  and 
no  other  variable. 

A  summary  of  the  three- wave  design  is  as  follows. 

•  Containment  waves  contain  the  propagation  of  sta- 
bilization  waves,  and  undo- containment  waves  con¬ 
tain  the  propagation  of  containment  waves. 

•  Each  containment  wave  is  associated  with  two  sta¬ 
bilization  waves:  the  old  stabilization  wave  which 
propagates  obsolete  information  and  which  is  con¬ 
tained,  and  the  new  stabilization  wave  which  de¬ 
activates  the  containment  wave  after  the  latter  has 
stopped  the  old  stabilization  wave. 

•  Each  contained  wave  (e.g.,  a  stabilization  wave)  sets 


Protocol  CPV.i 

Constant  d  :  set  of  node-ids 

dg ,  d<  •  dy .  Isyn  *  real 
Var  i.aspath,  i.tp  :  list  of  AS-ids 

i.j' .aspath,  i.j' .tp  ( j '  G  IM.i)  :  list  of  AS-ids 
i. ghost,  i.j' .ghost  ( j '  G  IM.i)  :  boolean 
i.t  :  real 
k  :  node-id 
Parameter  j  :  node-id 
Action 

(SW)  ::  S.i.j  V  R.i  V  R' .i - — - > 

if  S.i.j  —y  i.aspath  :=  aspath(i,  j)\ 
i. ghost  :=  i.j. ghost 

0 

R.i  —y  if  i  G  d  —y  i.aspath  :=  [i.as] 

0 

i  (fc  d  —y  i.aspath  :=  0 

fl; 

i.ghost  :=  false 

fl; 

i.tp  :=  0; 
i.t  :=  CLK.v, 

send  m(i.aspath, i.ghost, i.tp)  to  EX.i 

□ 

(CW)  ::  (-. i.ghost  A  JC.i)  V  TP.i  - ^ - t 

i.ghost  :=  true', 

do  S.i.k  —y  i.tp  :=  aspathli.k)  od; 
do  S' -i.k  —y  i.tp  :=  tp(i,k)  od; 
i.t  :=  CLK.v, 

send  m(i. ghost,  i.tp)  to  EX.i 

0 

(UW)  ::  LC.i  V  CC.i - — - > 

if  LC.i  —y  i.ghost  :=  false  fl; 
if  —y  i.tp  :=  0  fl; 
i.t  :=  CLK.v, 

send  m(i. ghost,  i.tp)  to  EX.i 

0 

*  *  * 

(SYN i)  ::  ( i.t  +  Isyn  <  CLK.i)  V  ( i.t  >  CLK.i)  — )■ 
i.t  :=  CLK.i ; 

send  m(i.aspath, i.ghost, i.tp)  to  EX.i 

0 

{SYN 2)  ::  rev  m  from  j  — y 
if  j  G  IM.i  —y 

update  i.j.aspath,  i.j.ghost,  and/or  i.j.tp 

fl 


Figure  3:  CPV:  Fault-Containing  Path  Vector  Routing 

the  boundary  of  its  corresponding  containing  wave 
(e.g.,  a  containment  wave)  such  that  the  latter  does 
not  propagate  beyond  where  the  former  has  reached. 

3.3  The  design  of  CPV 

The  protocol  CPV  is  shown  in  Figure  3,  where  the  pro¬ 
tocol  constants,  variables,  and  actions  for  each  node  i 
are  defined. 

Constants.  CPV  uses  five  constants:  d,  ds ,  dc.  du .  and 
Isyn ■  d  denotes  the  ID  of  the  destination  to  which  all 
other  nodes  in  the  network  need  to  maintain  a  route;  ds, 
dc,  and  du  are  used  to  control  the  propagation  speed 
of  stabilization  waves,  containment  waves,  and  undo- 
containment  waves  respectively;  Isyn  is  used  to  control 
the  frequency  of  information  synchronization  between 
neighboring  nodes. 
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To  contain  continuously  occurring  faults,  ds,  dc ,  and 
du  should  be  such  that  ds  >  ot-(dc  +  U),  dc  >  a-(du  +  U), 
and  du  >  0.  (Details  of  the  derivation  of  these  constants 
can  be  found  in  the  proof  of  Theorem  1.) 

Variables.  As  in  BGP,  node  i  maintains  its  AS-level 
path  to  d ,  denoted  as  i.aspath.  To  enable  containment 
waves,  i  uses  two  additional  variables:  i. ghost  and  i.tp. 
i. ghost  denotes  whether  or  not  i  is  involved  in  a  contain¬ 
ment  wave,  and  i.tp  denotes  the  predicted  route  which  i 
will  adopt  next.  To  coordinate  with  its  neighbors,  i  also 
maintains  a  copy  of  the  three  variables  for  each  of  its  im¬ 
port  neighbors  j',  denoted  as  i.j' .aspath,i.j' .ghost,  and 
i.j’.tp. 

For  convenience,  variable  i.t  is  used  to  record  the  time 
when  i  sends  a  message  to  its  export  neighbors  the  last 
time;  a  dummy  variable  k  is  also  used. 

Protocol  actions.  CPV  consists  of  five  actions,  one 
for  each  of  the  diffusing  waves  involved  in  CPV  and  two 
for  updating  information  between  neighbors.  We  briefly 
explain  each  action  by  its  category.  For  convenience,  we 
define  the  following  notations: 


aspath(i,  k) 
tp(i,  k ) 
rank(i,k) 

N(i ) 

IM.i 

EX.i 

CLK.i 


i.k.aspath,  if  i.as  =  k.as 

[i.as,i.k.aspath],  ii  i.as  y  k.as; 
i.k.tp,  if  i.as  =  k.as 

[i.as,i.k.tp\,  if  i.as  ^  k.as-, 

ma  x{rank(i.k.aspath),rank(i.k.tp)}; 
the  next-hop  of  i  on  its  route  to  d; 
the  set  of  import  neighbors  of  i: 
the  set  of  export  neighbors  of  i; 
the  current  clock  value  of  i. 


Stabilization  wave.  Stabilization  waves  are  imple¬ 
mented  by  action  S' IT.  SW  is  executed  when 

•  i  needs  to  propagate  a  stabilization  wave  from  an 
import  neighbor  j  (i.e.,  S.i.j  =  true),  i  needs  to 
reset  i.aspath  (i.e.,  R.i  =  true),  or  i  needs  to  reset 
i.tp  (i.e.,  R'.i  =  true);  and 

•  the  above  condition  has  continuously  held  for  the 
past  ds  time. 

Corresponding  to  the  intuitive  formulation  of  stabi¬ 
lization  waves  in  Section  3.2.B, 

.  S.i.j  = 


i  <t  d  A  j  £  IM.i  A  i.j.aspath  ^  0  A  i.as  ^  i.j.aspath  A 
(i.N(i).aspath  /0A  i.as  £  i.aspath  A  ->*.  ghost  => 

~<i.N(i). ghost)  A 

((-■ i.j. ghost  A  (VA:  :  k  £  IM.i  A  k  ±  j  => 

rank(i,k )  <  rank(i.j.aspath)))  V 
(i.j. ghost  A  (VA;  :  k  £  IM.i  A  k  J-  j  =>■ 

( i.k.aspath  y  %  =>  i.k.ghost)  A 
rank(i,k)  <  rank(i.j.aspath ))) 

)  A 

(i.as  £  i.aspath  V  (j  =  N(i)  A  i.aspath  J-  aspath(i,  j))  V 
(j  y  N(i)  f\rank(i.N(i).aspath)  <  rank(i.j.aspath))) 


•  R.i  = 

(j  £  d  A  i.aspath  J-  [d])  V 
(i  f  d  A  i.aspath  ^  0  A 
(VA;  :  k  £  IM.i  =>  (i.k.tp  =  0  V  i.as  £  i.k.tp) A 

(i.k.aspath  =  0  V  i.as  £  i.k.aspath))) 


•  R'.i  = 

i.tp  y  0  A 

(VA:  :  k  £  IM.i  =>  (i.k.tp  =  0  V  i.as  £  i.k.tp) A 

(i.k.aspath  =  0  V  i.as  £  i.k.aspath)) 

To  execute  SW,  i  updates  i.aspath  and  i. ghost  ap¬ 
propriately.  Since  the  execution  of  SW  means  fixing 
the  route  to  a  destination,  i  always  reset  i.tp  to  empty, 
signifying  that  i  will  not  change  route  any  more  unless 
faults  occur  again.  After  updating  its  state,  i  sends  the 
updated  state  to  its  export  neighbors. 

Containment  wave.  Containment  waves  are  imple¬ 
mented  by  action  CW.  CW  is  executed  when 

•  i  needs  to  join  a  containment  wave  (i.e,  -i. ghost  A 
JC.i  holds),  or  i  needs  to  update  i.tp  (i.e.,  TP.i  = 
true);  and 

•  the  above  condition  has  continuously  held  for  the 
past  dc  time. 

Corresponding  to  the  intuitive  formulation  of  contain¬ 
ment  waves  in  Section  3. 2. A, 

•  JC.i  = 

(i  £  d  A  i.aspath  ^  [d])  V 

(i  (j  d  A  ((3A;  :  S.i.k)  V  R.i  V 

(i.N(i). ghost  A  i.aspath  =  aspath(i,  N(i)))) 


•  TP.i  = 

(3k  :  S.i.k  A  i.tp  J-  aspath(i,  k))  V 
((i. ghost  V  JC.i)  A 

(3k  :  S' .i.k  A  i.tp  J-  tp(i,k)  A  i.aspath  J-  tp(i,k))) 
where  S'. i.j  = 

i  £  d  A  j  £  IM.i  A  i.j.tp  J-  0  A  i.as  ^  i.j.tp  A 

(VA:  :  k  £  IM.i  A  k  ^  j  =>  rank(i,  k)  <  rank(i.j.tp)) 

To  execute  CW,  i  sets  i. ghost  as  true  to  signify  that  it 
is  involved  in  a  containment  wave  and  will  change  route 
soon,  i  also  updates  i.tp  as  appropriate. 

[Improved  packet  forwarding  via  CW]  Since  contain¬ 
ment  waves  carry  information  on  relatively  fresher  net¬ 
work  state,  containment  waves  help  in  forwarding  pack¬ 
ets  in  the  presence  of  faults.  For  example,  when  i  is  the 
first  node  in  a  containment  wave  that  has  a  non-empty 
predicted  route  i.tp,3  i  can  use  i.tp  instead  of  i.aspath  to 
forward  packets,  since  i.tp  is  a  better  route  than  i.aspath. 
Similarly,  when  i  is  involved  in  a  containment  wave  but 
has  no  alternative  route  to  the  destination,4  i  can  imme¬ 
diately  stop  forwarding  packets  destined  for  the  destina¬ 
tion  to  avoid  wasting  network  resources. 

Undo -containment  wave.  Undo- containment  waves 
are  implemented  by  action  UW.  UW  is  executed  when 

•  i  is  involved  in  a  containment  wave  but  does  not 
need  to  change  route  any  more  (i.e.,  LC.i  =  true), 
or  the  part  of  the  state  of  i  related  to  containment 
waves  has  been  corrupted  (i.e.,  CC.i  =  true);  and 

3That  is,  (i. ghost  A  i.tp  y  0  A  -*i.N(i,  i.tp). ghost)  holds,  where 
N(i,i.tp)  denotes  the  next-hop  of  i  on  route  i.tp. 

4That  is,  i. ghost  =  true  and  i.tp  =  0. 
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•  the  above  condition  has  continuously  held  for  the 
past  du  time. 

Corresponding  to  the  intuitive  formulation  of  undo- 
containment  waves  in  Section  3.2.C, 

•  LC.i  = 

i. ghost  A 

((3k  :  S' .i.k  A  i.aspath  =  tp(i,  k))  V 
(- >JC.i  A  ->(3 k  :  S' -i.k))  V  (R' .i  A  ->R.t)) 

•  CCA  = 

(i.tp  #  0  A  -i(3fc  :  S.i.Jfc  V  S'.i.fc))  V 

(-• i.ghost  A  i.tp  ^  0  A  -lit'.i)  V  i.as  £  i.tp 

To  execute  [/IT.  i  resets  i.ghost  to  false,  and  if  there 
is  still  a  route  to  the  destination  (i.e.,  R'.i  =  false),  i 
resets  i.tp  to  empty. 

Note  that,  when  d  keeps  fail-stopping  and  rejoining 
at  high  frequencies,  it  is  critical  to  the  fault  contain¬ 
ment  that  the  reset  of  i.tp  be  performed  at  the  speed  of 
stabilization  waves  instead  of  that  of  undo-containment 
waves;  otherwise,  the  reset  of  i.aspath  could  potentially 
propagate  far  away. 

Information  update.  Actions  SYN i  and  SYN 2  en¬ 
able  neighboring  nodes  to  exchange  information  so  that 
information  consistency  is  guaranteed. 

•  Action  SYN l:  if  i  has  not  updated  its  state  with 
its  export  neighbors  for  more  than  Isyn  time  (i.e., 
t.i+Igyn  <  CLK.i )  or  if  i.t  is  corrupted  to  be  greater 
than  the  current  clock  value,  i  sets  i.t  to  its  cur¬ 
rent  clock  value,  and  sends  its  state  (i.e.,  i.aspath, 
i.ghost,  and  i.tp)  to  its  export  neighbors. 

•  Action  SYN 2:  when  i  receives  a  state  update  from 
an  import  neighbor  j,  i  updates  its  state  record  re¬ 
garding  j. 

3.4  Example  revisited 

We  reconsider  the  example  discussed  in  Section  3.1  by 
examining  how  the  network  behaves  if  CPV  is  adopted. 
For  simplicity  of  presentation,  we  assume  that  a  =  1, 
link  delay  is  a  constant  u,  processing  delay  is  negligible, 
and  the  propagation  speed  of  undo-containment  waves  is 
twice  that  of  containment  waves,  which  in  turn  is  twice 
that  of  stabilization  waves  (i.e.,  ds  =  2 dc  +  u  and  dc  = 
2 du  u) . 

When  d  fail-stops,  actions  SW  and  CW  are  enabled  at 
e  since  both  of  the  predicates  R.e  and  JC.e  hold.  Given 
that  dc  <  ds,  e  initiates  a  containment  wave  cw  1  (by 
executing  action  CW)  earlier  than  it  initiates  a  stabi¬ 
lization  wave  sw±.  Now  suppose  sw  1  has  reached  /  (i.e., 
action  SW  is  enabled  at  /)  and  cw  1  has  reached  g  (i.e., 
action  CW  is  enabled  at  g)  when  d  rejoins.  After  d  re¬ 
joins,  actions  SW  and  CW  are  enabled  at  e  again  since 
both  of  the  predicates  S.e.d  and  JC.e  hold.  After  dc  +  u 
time,  the  second  containment  wave  CW2  from  e  reaches 
/,  carrying  the  predicted  state  of  e  (i.e.,  e.tp  as  [d.os]); 
at  the  same  time,  cw  1  reaches  h.  When  cw 2  reaches 


/,  action  SW  becomes  disabled  at  /  (since  R.i  becomes 
false),  thus  sw  1  stops  at  /;  at  the  same  time,  action  UW 
becomes  enabled  at  /  (since  LC.f  holds),  while  CW  re¬ 
mains  disabled  at  e  (which  means  that  cw 2  stops  at  /). 
Since  undo- containment  waves  propagate  twice  as  fast 
as  containment  waves,  the  undo-containment  wave  uw± 
initiated  from  /  will  catch  up  with  cw  1  at  h,  after  which 
action  CW  becomes  disabled  at  h  and  cw  1  as  well  as  uw± 
stops  at  h. 

In  the  above  scenario,  therefore,  stabilization  waves 
propagate  to  /  the  farthest,  containment  as  well  as  undo- 
containment  waves  propagate  to  h  the  farthest,  and  node 
i  is  not  affected.  Moreover,  traffic  forwarding  is  only 
slightly  affected  in  the  sense  that  only  e  changes  its  route 
in  the  presence  of  faults  and  that  e  recovers  its  route  (via 
e.tp)  as  soon  as  d  rejoins. 

4  Continuous  containment  and 
local  stabilization 

Toward  establishing  the  foundation  for  studying  and  to¬ 
ward  precisely  characterizing  system  properties  in  the 
presence  of  continuously  occurring  faults,  we  formulate 
notions  related  to  fault  containment  and  local  stabi¬ 
lization:  perturbed  node,  contaminated  node,  pertur¬ 
bation  size,  contamination  range,  ./-"-containment,  and 
./-"-stabilization. 

We  regard  a  system  as  the  union  of  a  network  and 
the  protocols  running  on  it.  In  the  presence  of  faults, 
a  network  G  may  change  in  the  sense  that  its  topology 
or  routing  policy  function  changes,  where  the  topology 
of  G  is  the  subgraph  G'(y',E')  of  G(V,E)  such  that 
V'  =  {/  :  i  S  V  A  i  is  up}  and  E'  =  {(i,j)  :  i  €  V'  A  j  £ 
V'A (i,  j)  £  EA(i,j)  is  up}.  To  reflect  changes  in  network 
topology  and  routing  policy  function,  we  regard  the  state 
of  a  system  as  the  union  of  the  network  topology,  the 
routing  policy  function,  and  the  state  of  all  the  up  nodes, 
with  the  state  of  a  node  being  the  values  of  the  variables 
maintained  at  the  node.  At  a  system  state  q,  the  network 
topology  is  denoted  by  G.q(V.q,E.q),  and  the  state  of  a 
node  j  is  denoted  as  j.q. 

We  characterize  the  behavior  of  a  system  in  the  pres¬ 
ence  of  faults  by  the  system  history.  A  system  history 
7i  is  either  a  finite  sequence  qo,  (ei,fi),  qi,  ( e2,^2 ),  ■  ■  ■ , 

qn,  or  an  infinite  sequence  qo,  (ei,ti),  qi,  (e2,t2),  ■■■, 
qk-i,  ( ek,tk ),  qki  ■  ■■,  of  alternating  system  states  (i.e. 

qo,  q±, . . .)  and  events  (i.e.  e±,  e2,  •  ■  ■),  where 

•  An  event  is  either  the  execution  of  a  protocol  action 
or  the  occurrence  of  a  fault; 

•  For  every  k  >  1,  tk  <  tk+ 1,  and  each  state  transition 
qk-i,(^k,tk),Qk  means  that  ek  at  time  tk  changes 
the  system  state  from  qk-i  to  qk',  and 

•  For  any  two  pairs  (e*,  tk)  and  (e# ,  ty )  in  hi  (k  k'), 
if  ek  and  e#  occur  at  the  same  node,  then  tk  tk> 
(i.e.,  at  most  one  event  can  occur  at  a  node  at  any 
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time). 

Ti  is  a  finite  sequence  only  if  it  ends  with  a  state  qn  such 
that  there  is  no  action  enabled  at  qn  and  no  fault  occurs 
after  the  system  reaches  qn. 

A  subsequence  a  of  a  system  history  H  is  called  a 
history  segment  if  a  starts  and  ends  with  a  state.  A 
history  segment  /?  is  called  a  history  prefix  if  ft  starts 
with  the  initial  state  qo.  For  convenience,  we  denote  as 
?i(q)  a  history  prefix  ending  with  a  state  q.  A  history 
segment  7  starting  at  a  state  qk  is  called  a  computation 
starting  at  qk  if  7  is  the  suffix  of  7i  starting  at  qk  and 
every  event  in  7  is  the  execution  of  a  protocol  action  (i.e., 
no  fault  occurs  in  7). 

Given  a  network,  a  protocol  specification  defines  a 
set  of  legitimate  states.  For  example,  given  a  net¬ 
work  topology,  a  routing  policy  function,  and  a  desti¬ 
nation,  the  specification  of  BGP  determines  a  set  of  le¬ 
gitimate  states,  each  of  which  specifies  the  route  of  every 
node.  For  a  self-stabilizing  protocol,  each  computation 
Ck  starting  at  an  arbitrary  state  qk  determined  a  con¬ 
verged  state  L(qk,Ck)-  Whenever  a  fault  changes  a  sys¬ 
tem  state  from  qk  to  the  instance  of  computation 

changes  in  the  sense  that  the  computation  Ck+i  start¬ 
ing  at  qk- 1-1  is  different  from  Ck-  Then,  given  a  state 
qk  with  a  history  prefix  7i(qk),  we  regard  as  a  proto¬ 
col  execution  £(qk)  a  set  of  computations  each  of  which 
specifies  a  computation  C(qk'  ,£(qk ))  for  a  different  state 
q^  in  7i(qk)  that  is  either  the  initial  state  or  a  state 
reached  immediately  after  a  fault  occurs.  Then,  given 
an  arbitrary  state  qk  and  an  arbitrary  protocol  execu¬ 
tion  £(qk),  we  define  the  stabilization-set  of  qk  under 
£(qk),  denoted  by  Ss(qk,£(qk)),  as  the  set  of  nodes  that 
need  to  change  state  in  order  for  the  network  to  stabi¬ 
lize  from  qk,  i.e.,  Ss(qk,£(qk ))  =  {j  ■  j  £  V.qk  A  j.qk 
j.L(qk,C(qk,£(qk)))}.  (Of  course,  if  qk  is  a  legitimate 
state,  Ss(qk,£(qk ))  =  0-) 

If  an  event  e  occurring  at  time  tk  changes  the  sys¬ 
tem  state  from  qk- 1  to  qk  under  a  protocol  execution 
£(qk ),  we  define  the  corruption  set  of  e  at  tk,  denoted 
by  cpt(e,tk,£{qk )),  as  the  set  of  nodes  that  need  not 
change  state  in  order  for  the  network  to  stabilize  from 
qk~  1,  but  need  to  change  state  in  order  for  the  network  to 
stabilize  from  qk,  i.e.,  cpt(e,tk,£(qk ))  =  Ss(qk,£(qk ))  \ 
Ss(qk-i,£(qk))-  In  addition,  if  e  is  not  a  state  corrup¬ 
tion,  we  define  the  correction  set  of  e  at  tk,  denoted 
by  cct(e,tk,£(qk)),  as  the  set  of  nodes  in  V.qk  that 
need  to  change  state  in  order  for  the  network  to  sta¬ 
bilize  from  qk- 1,  but  need  not  change  state  in  order  for 
the  network  to  stabilize  from  qk,  i.e.,  cct(e,tk,£{qk))  = 
(Ss(qk-i,£(qk))  \  Ss(qk,£(qk)))  H  V.qk.  If  e  is  a  state 
corruption,  we  define  cct(e,tk,£(qk))  as  0. 
Perturbation  vs.  contamination.  For  every  node 
j  £  cpt(e,tk,£{qk )),  we  regard  j  as  perturbed  by  e  if 
e  is  a  fault,  and  we  regard  j  as  contaminated  via  e  if 
e  is  the  execution  of  a  protocol  action;  for  every  node 
j  £  cct(e,tk,£{qk )),  we  regard  j  as  corrected  by  e.  For 


instance,  in  the  example  discussed  in  Section  3.1,  when 
d  fail-stops,  are  perturbed  by  the  fail-stop  of  d; 

when  d  rejoins  before  /  withdraws  its  route,  f,...,i  are 
corrected  by  the  join  of  d;  when  BGP  is  used,  f,...,i 
continue  to  withdraw  their  routes  after  d  rejoins,  thus 
/,...,*  are  contaminated  via  BGP  actions. 

Given  a  system  state  qk  with  a  history  prefix  ' H(qk ) 
and  a  protocol  execution  £{qk),  a  node  i  is  perturbed  at 
qk  if  either  of  the  following  conditions  hold: 

•  qk  is  the  initial  state  (i.e.,  k  =  0),  and  i  needs  to 
change  state  in  order  for  the  network  to  stabilize. 

•  i  has  been  perturbed  by  a  fault  at  some  point,  and 
neither  i  has  been  corrected  by  a  fault  nor  has  the 
network  reached  a  legitimate  state  ever  since. 

A  node  i  is  contaminated  at  qk  if  i  has  been  contaminated 
at  some  point  and  has  not  been  corrected  ever  since. 
Formally, 

Definition  1  (Perturbed  node)  Given  a  protocol  ex¬ 
ecution  £{qk),  and  a  network  state  qk  with  history  prefix 
H(qk)  (k  >  0),  a  node  i  (i  £  V.qk)  is  perturbed  at  qk  if 
and  only  if 

•  i  £  Ss(qk,£(qk)), 

when  k  =  0 

•  PT{i,qo,qk,U(qk),£{qk))  =true, 

when  k  >  0A->(3fc'  :  qk<  £  H(qk)/\qk'  £  Q^qk1)) 

•  PT(i,LL(qk,H(qk)),qk,H(qk),£(qk))  =  true, 

when  k  >  0A(3 k'  :  qk<  £  Tt{qk)Aqk'  £  Ql^w)) 

where 

PT{i,qm,qn,n{qn),£{qk))  = 
qm  G  ((/,})  A 

((e„  is  a  fault  Ai  £  cpt(en,tn,£(qk)))  V 
(3fc'  :  m  <  k’  <  n  A  i  is  perturbed  at  qk<  A 
-'(3/  :  k!  <  j'  <  n  A  (ef ,  tr )  £  H(qn)  A 
eji  is  a  fault  Ai  £  cct(eji  ,tj> )))) 
LL(qk,‘H(qk ))  =  qj  such  that 
qj  e  H{qk)  A  qj  £  QL(qj)  A 
—>(3  /  =  j  <  j'  <  k  A  qy  £  U{qk)  A  qr  £  QL{qf)) 
Q l  (qj )  =  the  set  of  legitimate  states  corresponding 
to  the  network  topology  and  routing  policy 
function  at  qj . 

Definition  2  (Contaminated  node)  Given  a  proto¬ 
col  execution  £  (qk ) ,  and  a  network  state  qk  with  history 
prefix 'H(qk)  (k  >  0),  a  node  i  (i  £  V.qk)  is  contaminated 
at  qk  if  and  only  if 

•  CT(i,q0,qk,n(qk),£(qk))  =true, 

when  k  >  0A-i(3fc/  :  qk’  £  7i(qk)  Aqk’  £  Qx(qk')) 

•  CT(i,LL(qk,7i(qk)),qk,'H(qk),£(qk))  =true, 

when  k  >  0A(3 k'  :  qk<  £  U{qk)Aqk'  £  Q^Qk1)) 

where 

CT(i,qm,qn,7i(qn),£{qk ))  = 
qm  £  H(qn) A 

((en  is  not  a  fault  A  i  £  cpt{en,tn,£{qk )))  V 
(3 k'  :  m  <  k'  <  n  A  i  is  contaminated  at  qk'  A 
{r.f  :  k'  <  j'  <  n  A  (e^ ,  tj> )  £  H(qn)  A 
i  £  cct(eji  ,tji)))) 
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LL(qk,7i(qk))  =  Qj  such  that 
qj  e  H{qk)  A  qj  £  QL{qj )  A 
-<3/  :  j  <  j'  <  k  A  qj'  £  U{qk)/\qj>  €  Ql^j')) 
Ql(Qj)  =  the  set  of  legitimate  states  corresponding 
to  the  network  topology  and  routing  policy 
function  at  qj . 

Intuitively,  these  definitions  imply  that,  a  perturbed 
node  remains  perturbed  until  it  is  corrected  by  a  fault  or 
the  network  reaches  a  legitimate  state;  a  contaminated 
node  remains  contaminated  until  it  is  corrected  by  ei¬ 
ther  a  fault  or  the  execution  of  a  protocol  action.  For 
instance,  in  the  example  discussed  in  Section  3.1,  at  the 
state  immediately  after  d  fail-stops,  are  all  per¬ 

turbed;  at  the  state  immediately  after  d  rejoins,  only  e 
is  perturbed,  since  /,...,  i  are  corrected  by  the  rejoin¬ 
ing  of  d ;  at  the  state  where  /  withdraws  its  route,  /  is 
contaminated. 

The  number  of  perturbed  nodes  at  a  system  state, 
which  we  define  as  the  perturbation  size,  reflects  the 
severity  of  the  impact  that  faults  have  on  the  system. 
Formally, 

Definition  3  (Perturbation  size)  Given  a  system 
state  qk  with  a  history  prefix  H(qk)  and  a  protocol 
execution  £(qk),  the  perturbation  size  at  qk,  denoted 
by  'P(clki,hL(qk),£(qk))>  is  the  number  of  nodes  that  are 
perturbed  at  qk.  That  is,  V(qk,'H(qk),£(qk))  =  \ {i  :  i  £ 
V.qk  Ai  is  perturbed  at  qk} |. 

A  set  5  of  nodes  are  contiguous  at  a  system  state  q 
if  5  C  V.q  and  the  subgraph  of  G.q(V.q,E.q)  on  5  is 
connected,  i.e.,  the  graph  G'(V' ,E')  is  connected,  where 
V'  =  S  and  E'  =  :  i  £  S  A  j  £  S  A  (i,  j)  £  E.q }. 

We  regard  a  maximal  set  of  perturbed  nodes  that  are 
contiguous  as  a  perturbed  region.  We  also  regard  a  con¬ 
taminated  node  i  as  being  contaminated  by  a  perturbed 
region,  via  the  execution  of  a  protocol  action  a,  if  a  is 
executed  at  i  because  of  the  state  changes  at  a  node  j 
that  is  either  in  or  contaminated  by  the  perturbed  re¬ 
gion.  Then,  given  a  perturbed  region  Sp  at  a  state  qk, 
the  maximum  distance  from  the  nodes  contaminated  by 
Sp  to  Sp,  which  we  define  as  the  contamination  range  of 
Sp  at  qk,  reflects  the  severity  of  fault  propagation  from 
Sp  at  qk-  Formally, 

Definition  4  (Contamination  range)  Given  a  per¬ 
turbed  region  Sp  at  a  state  qk,  the  contamination  range 
of  Sp  at  qk,  denoted  by  R(Sp,qk),  is 

max  hops(i,  Sp,qk) 

iESc 

where 

Sc  =  {i:i  £  V.qk  A 

i  is  contaminated  by  Sp}, 
hops(i,  Sp,qk)  =  min j£Sp  hops{i,  j,qk), 
hops(i,j,qk)  =  the  number  of  hops  in  a  shortest 
path  between  i  and  j  in  G.qk. 

To  prevent  fault  propagation  and  to  increase  the  sta¬ 
bility  as  well  as  the  availability  of  networks  in  the  pres¬ 


ence  of  continuously  occurring  faults,  it  is  desirable  that, 
at  every  state,  the  contamination  range  of  each  per¬ 
turbed  region  remain  bounded  relative  to  the  size  of  the 
perturbed  region.  Formally,  this  property  is  character¬ 
ized  as 

Definition  5  (JF-containment)  A  system  is  T- 
containing  if  and  only  if 

For  every  perturbed  region  Sp  at  an  arbi¬ 
trary  system  state  qk  (k  >  0),  R(Sp,qk)  = 
0(E(\Sp\)),  where  T  is  a  function. 

Besides  containing  faults  locally  around  where  they  oc¬ 
cur,  it  is  desirable  that,  once  faults  stop  occurring  (either 
indefinitely  or  for  a  long  enough  period),  the  network 
stabilizes  quickly  and,  intuitively,  within  time  depend¬ 
ing  on  the  perturbation  size.  Formally,  this  property  is 
characterized  as 

Definition  6  (JF-stabilization)  A  system  is  T- 
stabilizing  if  and  only  if 

Starting  at  an  arbitrary  state  qk  with  an  arbi¬ 
trary  history  prefix  H(qk)  and  an  arbitrary  pro¬ 
tocol  execution  £(qk),  the  system  computation 
is  guaranteed  to  reach  a  legitimate  state  within 
0(J:('P(qk,'H(qk),£(qk))))  time  in  the  absence 
of  faults,  where  T  is  a  function. 

5  Analysis  of  CPV 

Given  a  network  topology  G'(V',  E')  and  a  routing  policy 
function  P' ,  we  let 

C  =  (Vi  :  i  E  V1  =>•  -4. ghost  A  i.tp  =  0  A  LH.i) 

where  LH.i  = 

(i  E  d  =>  i.aspath  =  0)  A 

(*  f:  d  =>  (( d  C  V'  =>  i.aspath  =  aspath(i,hr(i,P')))  A 
(d  V'  =>-  i.aspath  =  0))) 

with  hr(i,P')  being  the  highest-ranked  neighbor  of  i,  i.e. 

(yk  :  k  G  IM.ihk  ^  hr(i,P')  =>■  rank(i.k.aspath)  < 

rank(i.hr(i,  P').aspath )) 

Then,  every  state  in  C,  is  a  state  where  every  up  node  in 
the  network  has  found  its  best  route  to  the  destination. 
Thus  every  state  in  C  is  a  legitimate  state. 

To  analyze  the  properties  of  CPV,  we  first  prove  that 
CPV  is  self-stabilizing,  then  we  prove  that  CPV  contains 
continuously  occurring  faults  and  locally  stabilizes. 
Lemma  1  (Self-stabilization)  Starting  at  an  arbi¬ 
trary  state,  every  computation  of  a  system  where  CPV 
is  used  is  guaranteed  to  reach  a  state  in  C. 

Proof:  To  prove  this  lemma,  we  first  prove  the  sta¬ 
bilization  property  for  the  common  case  where  the  SPF 
policy  is  used;  then  we  prove  the  case  where  general  route 
ranking  policy  (which  may  not  be  the  SPF  policy)  is  used 
(by  the  help  of  existing  results  on  the  convergence  prop¬ 
erties  of  Simple  Path  Vector  Protocols  [?]). 

The  SPF  policy. 
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First,  we  introduce  the  concept  of  “computation 
rounds”  which  will  be  used  in  the  proof.  A  system  com¬ 
putation  H  can  be  regarded  as  a  sequence  of  rounds.  A 
round  is  a  minimal  computation  segment  7  that  starts 
at  a  state  qk  (k  >  0)  and,  in  7,  (i)  every  up  node  that 
has  an  action  a  continuously  enabled  from  some  time  t' 
to  ( t'+d.a ),  where  t'  <  tk  and  ( t'+d.a )  >  tk,  executes  at 
least  one  action,  and  (ii)  if  a  message  is  sent  to  a  node  i, 
the  action  that  receives  the  message  must  be  executed  at 
i.  (We  assume  to  is  the  time  when  Tt  starts.)  For  CPV, 
each  round  of  a  computation  lasts  at  most  ( ds  +  U )  time 
in  length. 

When  the  SPF  policy  is  used,  we  prove  that 
[Claim  0]  Starting  at  an  arbitrary  state,  every  compu¬ 
tation  of  a  system  where  CPV  and  the  SPF  policy  are 
used  is  guaranteed  to  reach  a  state  in  £  within  0(|V'|) 
time. 

To  prove  Claim  0,  we  analyze  the  two  different  cases 
that  can  exist:  when  d  is  up  (i.e.,  d  C  V '),  and  when  d 
is  down  (i.e.,  d  ^  V’). 

[Claim  0.0]  Starting  at  an  arbitrary  state  when  d  is  up 
(i.e.,  d  C  V'),  every  computation  of  a  system  where  CPV 
and  the  SPF  policy  are  used  is  guaranteed  to  reach  a 
state  in  £  within  O(D)  time,  where  D  is  the  number  of 
hops  in  the  longest  shortest  path  from  a  node  not  in  d 
to  d  in  G'. 

In  this  case,  we  prove  the  claim  by  the  “convergence 
stair”  approach  used  in  [2]  and  some  other  works.  That 
is,  That  is,  we  exhibit  a  finite  sequence  of  state  predicates 
£.0,  £.1, . . . ,  C.D  such  that 

(i)  Starting  at  an  arbitrary  state,  the  system  is  guar¬ 
anteed  to  reach  a  state  in  £.0. 

(ii)  For  each  l  such  that  0  <  l  <  D: 

C.l  is  closed  under  system  execution;  that  is,  once 
C.l  holds  in  an  arbitrary  system  computation,  it 
continues  to  hold  subsequently. 

(iii)  For  each  l  such  that  0  <  l  <  D: 

Upon  starting  at  an  arbitrary  state  in  C.l,  the 
system  is  guaranteed  to  reach  a  state  in  C.(l  +  1) 
within  a  constant  amount  of  time. 

(iv)  C.D  =  £ 

For  our  protocol,  £.0,  £.1, . . . ,  C.D  are  defined  as  fol¬ 
lows, 

•  £.0  =  (Vi  :  i  G  d  =>-  i.aspath  =  0) 

•  For  0  <  l  <  D: 

C.(l  +  1)  =  C.lA 

(Vi  :  i  £  V'  A  hops(i,  d,  G’)  =  l  +  1  => 

-1  i. ghost  A  i.tp  =  0  A  LH.i) 

where 

hops(i,  d,G ')  =  the  number  of  hops  in  the  shortest  path 
from  i  to  d  in  G1 . 

We  prove  the  stabilization  of  £  by  proving  the  indi¬ 
vidual  elements  of  the  “convergence  stair”  method: 

(i)  Starting  at  an  arbitrary  state  qo,  the  system  is  guar¬ 
anteed  to  reach  a  state  in  £.0  within  constant  (i.e., 
ds)  time. 

If  £.0  holds  at  state  qo ,  we  are  done. 


If  £.0  does  not  hold  at  qo,  then  R.i  must  hold 
for  some  node  i  £  d.  For  each  of  such  node  i, 
therefore,  action  SW  is  enabled  at  qo  and  will  be 
executed  within  ds  time. 

Therefore,  starting  at  an  arbitrary  state  qo,  the 
system  is  guaranteed  to  reach  a  state  in  £.0  within 
ds  time. 

(ii)  For  each  l  such  that  0  <  l  <  D:  C.l  is  closed  under 
system  execution. 

To  prove  this  claim,  we  first  prove  that,  starting 
at  state  qo,  all  invalid  routes  that  are  of  hop- length 
l'  are  removed  within  l'  rounds  of  system  compu¬ 
tation.  This  comes  from  the  fact  that,  for  every 
such  invalid  route  r,  nodes  in  the  ASes  of  r  exe¬ 
cute  action  S' TV  which  removes  subpath  r  gradu¬ 
ally.  Therefore,  this  sub-claim  is  easily  proved  by 
induction  on  the  number  of  hops  from  an  AS  in  r 
to  d.  (For  conciseness,  we  skip  the  detail  here.) 

Therefore,  once  the  system  reaches  a  state  in  C.l, 
neither  action  SW  nor  action  CW  will  be  enabled 
for  nodes  whose  minimum  AS-level  hop  distance  to 
d  is  less  than  or  equal  to  l  (i.e.,  hops(i,  d,  Gr)  <  l 
for  each  of  such  node  i),  as  a  result  of  which  no 
such  node  will  change  state  any  more.  Thus,  once 
the  system  reaches  a  state  in  C.l,  the  system  state 
remains  in  C.l  subsequently. 

(iii)  For  each  l  such  that  0  <  l  <  D:  Upon  starting  at 
an  arbitrary  state  in  C.l,  the  system  is  guaranteed 
to  reach  a  state  in  C.(l  +  1)  within  (ds  +  U)  time. 

To  prove  this,  we  only  need  to  prove  that  for  a 
node  i  where  hops(i,  d,  G'J  =  1  +  1  but  ->*. ghost  A 
i.tp  A  LH.i  does  not  hold  when  the  system  is  at 
state  C.l,  =i.ghost  A  i.tp  A  LH.i  will  become  hold¬ 
ing  within  (ds  +  U)  time.  This  is  true  because,  if 
H.  ghost  A  i.tp  A  LH.i  does  not  hold  when  the  sys¬ 
tem  is  at  state  C.l,  S.i.k  =  true  for  some  import 
neighbor  k  of  i  where  hops(k,d,G')  =  Z;  thus,  ac¬ 
tion  S' TV  is  enabled  at  i  and  will  be  executed  within 
ds  time;  given  that  it  takes  up  to  U  time  for  the 
new  state  of  k  to  reach  i,  i  executes  action  STV 
within  ds  +  U  time  after  the  system  reaches  a  state 
in  C.l,  by  which  LH.i  becomes  true,  i. ghost  is  set 
to  false,  and  i.tp  is  set  to  0. 

(iv)  C.D  =  C 

This  trivially  holds  since  the  maximum 
minimum-hop-distance  from  a  node  in  V'  to 
d  is  D. 

Therefore,  Starting  at  an  arbitrary  state  when  d  is 
up  (i.e.,  d  C  V'),  every  computation  of  a  system  where 
CPV  and  the  SPF  policy  are  used  is  guaranteed  to  reach 
a  state  in  £  within  0(D)  time. 

[Claim  0.1]  Starting  at  an  arbitrary  state  qo  when  d  is 
done  (i.e.,  d  ^  V’),  every  computation  of  a  system  where 
CPV  and  the  SPF  policy  are  used  is  guaranteed  to  reach 
a  state  in  £  within  O(jV'J)  time. 

Claim  0.1  is  easily  proved  by  following  the  same  ap- 
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proach  as  used  in  proving  Claim  0.0  and  in  [?].  That  is, 
by  considering  each  route  r  that  exists  at  state  qo,  then 
nodes  in  the  ASes  of  r  execute  action  SW  which  removes 
subpath  of  r  gradually.  Therefore,  by  induction  on  the 
number  of  hops  from  an  AS  in  r  to  d,  we  see  that,  starting 
at  qo,  there  will  be  no  route  in  the  system  within  0( \V'\) 
rounds  of  computation,  since  the  maximum  hop-length 
of  a  loop- free  route  is  \V'\,  and  every  looping  route  is 
removed  within  constant  time  by  executing  action  SW. 
Therefore,  starting  at  state  qo,  the  system  will  reach  a 
state  where  no  node  has  a  route  to  d  (i.e.,  a  state  in  T) 
within  0(  |V'|)  time. 

Having  proved  the  lemma  for  the  case  where  the  SPF 
policy  is  used,  we  prove  the  lemma  as  follows  for  cases 
where  non-SPF  policies  are  used. 

Non-SPF  policies. 

To  prove  the  convergence  of  CPV  from  an  arbitrary 
state  to  a  state  in  T  when  route  ranking  policies  other 
than  the  SPF  policy  is  used,  we  first  prove  that 
[Claim  1]  starting  at  an  arbitrary  state  qk,  the  system 
reaches  a  state,  within  constant  time,  where  the  state 
related  to  containment  waves  and  stabilization  waves  is 
consistent  with  each  other,  that  is,  a  state  in  Co,  with  To 
defined  as  (Vi  :  CNSTO.i  A  CNSTl.i),  where  CNSTO.i 
is  defined  as 

i.tp  ±  0  => 

(i. ghost  A 
(i TP.i  => 

(3k  :  (S.i.k  A  i.tp  =  aspath(i,  k))  V  (S' .i.k  :  i.tp  =  tp(i,  k))))) 
and  CNSTl.i  is  defined  as 

i. ghost  =>-  JC.i  V  (3k  :  S'  .i.k) 

To  prove  Claim  1,  we  first  prove  as  follows  that,  start¬ 
ing  at  an  arbitrary  state  qk,  the  system  reaches  a  state 
where  (Vi  :  CNSTO.i )  holds: 

•  Starting  at  an  arbitrary  state  qk,  the  system  reaches 
a  state  where  (Vi  :  i.tp  =>  i. ghost)  holds  within  con¬ 
stant  time. 

For  a  node  i  where  i.tp  ^  0A~i i. ghost  holds,  action 
UW  is  enabled  if  R'.i  =  false  (since  CC.i  holds) 
and  action  S' IT  is  enabled  if  R'.i  =  true.  There¬ 
fore,  i  executes  UW  within  du  time  or  executes  S' IT 
within  ds  time,  as  a  result  of  which  i.tp  is  set  to  0 
and  thus  i.tp  =>  i. ghost  becomes  hold  for  i. 

•  Starting  at  an  arbitrary  state  where  (Vi  :  i.tp  => 
i. ghost)  holds,  the  system  reaches  a  state  where  (Vi  : 
CNSTO.i)  holds  within  constant  time. 

For  a  node  i  where  i.tp  ^  0  A  -TP.i  A  -*{3 k  : 

( S.i.kAi.tp  =  aspath(i,  k))V(S'.i.kAi.tp  =  tp(i,  k ))) 
holds,  action  UW  is  enabled  at  i  since  CC.i  =  true. 
Therefore,  i  executes  UW  within  du  time,  as  a  result 
of  which  i.tp  =  0  and  thus  CNSTO.i  holds. 

Then  we  prove  that,  starting  at  an  arbitrary  state 
where  (Vi  :  CNSTO.i)  holds,  the  system  reaches  a  state 
in  (Vi  :  CNSTO.i  A  CNSTl.i)  within  constant  time: 


For  a  node  i  where  CNSTO.i  A  i, ghost  A  JC.i  A 
-i(3 k  :  S'.i.k)  (i.e.,  CNSTO.i  =  true  but 

CNSTl.i  =  false),  action  UW  is  enabled  since 
LC.i  =  true.  Therefore,  i  executes  UW  within 
du  time,  as  a  result  of  which  CNSTO.i  A  CNSTl.i 
holds. 

Therefore,  Claim  1  holds;  then  we  prove  that 
[Claim  2]  starting  at  an  arbitrary  state  q'kinCo,  the  sys¬ 
tem  reaches  a  state  in  C  within  0(r(|T'|))  time,  where 
r(|T'|)  depends  on  the  route  ranking  policy  used  in  a 
network  and  is  the  diameter  of  the  timed  dispute  digraph 
of  the  system  [?]. 

To  prove  Claim  2,  we  first  prove  that,  starting  at  an 
arbitrary  state  in  Cq  where  C  does  not  hold,  action  SW 
will  be  executed  at  a  node  within  constant  time.  In  a 
state  To  where  T  does  not  hold,  there  can  be  two  cases: 

•  There  is  no  node  i  such  that  i. ghost  =  true: 

In  this  case,  there  must  be  a  node  i  such  that 
(3fc  :  S.i.k)VR.i  holds  (otherwise,  the  system  would 
be  in  state  T).  Therefore,  node  i  will  execute  ac¬ 
tion  SW  within  ds  time,  unless  another  neighboring 
node  executes  S' IT  which  disables  (3  k  :  S.i.k)  V  R.i. 
Either  way,  there  is  at  least  a  node  executing  SW 
within  ds  time. 

•  There  is  at  least  one  node  i  such  that  i. ghost  =  true: 

At  a  state  in  To  where  there  is  some  node  i  such 
that  i. ghost  =  true,  there  must  be  a  node  j  such 
that  (3fc  :  S.j.k)VR.j  holds  (otherwise,  there  would 
be  no  node  in  any  containment  wave).  Therefore, 
node  j  will  execute  action  SIT  within  ds  time,  un¬ 
less  another  neighboring  node  executes  S' IT  which 
disables  (3fc  :  S.j.k)  V  R.j.  Either  way,  there  is  at 
least  a  node  executing  SW  within  ds  time. 

Thus,  starting  at  a  state  q'k  in  To,  there  will  be  an 
action  SW  executed  within  every  constant  time  until 
the  system  reaches  a  state  in  T.  By  theorems  in  [?],  if 
an  action  SW  is  executed  at  time  t,  which  leads  to  a  new 
system  state  qk" ,  then  there  must  exist  a  path  from  q'k  to 
qk n  in  the  Timed  Dispute  Graph  TDD  ((?')  of  the  system. 
Therefore,  starting  at  a  state  qkinCo,  the  system  reaches 
a  state  in  T  within  0(r(|V'|))  time,  where  r(jT/|)  is  the 
diameter  of  the  timed  dispute  digraph  of  the  system. 

Therefore,  Claim  2  holds;  to  finish  the  proof,  we  only 
need  to  prove  that 

[Claim  3]  once  a  system  reaches  a  state  in  T,  the  system 
remains  in  T  in  the  absence  of  faults. 

Claim  3  trivially  holds,  since  none  of  actions  SW, 
CW,  and  UW  is  enabled  at  any  node  at  a  state  in 

T. 

Therefore,  Lemma  1  holds  for  cases  where  policies 
other  than  the  SPF  policy  are  used.  That  is,  starting  at 
an  arbitrary  state,  every  computation  of  a  system  where 
CPV  is  used  is  guaranteed  to  reach  a  state  in  T  within 
0(r(|V'|))  time,  where  r(jT/j)  is  the  diameter  of  the 
Timed  Dispute  Digraph  TDD(G')  of  the  system  (which 
depends  on  the  route  ranking  policy  used). 
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□ 

For  cases  where  faults  keep  occurring  at  high  frequen¬ 
cies,  we  have 

Theorem  1  (Continuous  containment)  In  a  system 
where  CPV  is  used,  the  contamination  range  R(SP,  qk) 
of  every  perturbed  region  Sp  at  an  arbitrary  state  qk  is 
0(|5P|).  That  is,  a  system  where  CPV  is  used  is  T- 
containing,  with  T  being  a  linear  function. 

Proof  :  To  prove  this  theorem,  we  first  consider  the 
case  where  there  is  only  one  perturbed  region  in  the  sys¬ 
tem,  then  we  consider  the  case  where  multiple  perturbed 
regions  co-exist. 

[Single  perturbed  region]  In  CPV,  for  every  node 
i,  only  action  SW  and  CW  could  propagate  the  ef¬ 
fect  of  faults,  since  only  stabilization  and  containment 
waves  propagate  the  state  changes  at  a  node  away  from 
the  node  (whereas  undo- containment  waves  only  undo 
mistakenly-propagated  containment  waves,  and  undo- 
containments  waves  do  not  maintain  any  variable  at  all). 
Therefore,  to  prove  this  theorem  when  there  is  a  sin¬ 
gle  perturbed  region,  we  need  to  prove  that,  for  any 
perturbed  region  Sp,  its  contamination  range  is  always 
0(r(|Sp|))  irrespective  the  execution  and  the  system  his¬ 
tory  prefix.  There  are  two  cases  that  can  propagate  the 
impact  of  the  perturbed  region  during  protocol  execution 
that  tries  to  stabilize  the  network  from  the  perturbed 
state  by  the  faults;  we  prove  that  in  neither  case  will  the 
propagation  be  unbounded. 

•  Case  one:  there  is  a  node  i  that  has  a  neighbor  in 
Sp,  and  i  has  just  initiated  a  containment  wave  (e.g., 
due  to  fast  state  oscillation)  when  i  turns  out  not  to 
initiate  any  stabilization  wave. 

For  every  such  node  i,  action  UW  is  enabled  since 
LC.i  =  true.  Therefore,  an  undo-containment  wave 
will  be  initiated  that  propagates  along  the  same  di¬ 
rection/paths  as  the  containment  wave,  as  shown  in 
Figure  ??. 


Figure  4:  Containment  wave  in  the  absence  of  accompa¬ 
nying  stabilization  wave 

Because  the  propagation  speeds  of  different  dif¬ 
fusing  waves  are  controlled  by  introducing  delays  in 
action  execution  when  a  node  schedules  its  enabled 
actions,  and  there  is  difference  between  clock  rates 
at  different  nodes,  dc  >  a-(du+U)  should  hold  in  or¬ 


der  for  the  undo-containment  wave  to  catch  up  with 
the  containment  wave.  Under  this  condition,  the 
farthest  distance  dt  (in  terms  of  the  number  of  hops) 
the  containment  wave  could  propagate  satisfies  the 
following  formula:  ( U  +  du)  xdt  +  C 0  =  ( L  +  dc)xdt 
(CO  is  the  delay  in  i  initiating  the  undo-containment 
wave),  thus  dt  =  d  _dc/L_jj  and  is  a  constant. 

Therefore,  the  theorem  holds  in  this  case. 

•  Case  two:  there  is  a  node  i  that  has  a  neighbor 
in  Sp,  and  i  has  initiated  a  containment  wave  CW 0 
because  i  needs  to  initiate  a  stabilization  wave  SWO 
by  protocol  CPV  (see  Figure  ??). 


SW0Ji //--  cwi 

CWO  /iff  S  W1 

/  ®  UW1 

* 

Figure  5:  Consistent  containment  and  stabilize  waves 

In  the  presence  of  continuously  occurring  faults, 
there  are  two  subcases: 

—  If  i  needs  to  change  state  after  another  fault 
/  occurs,  then  every  node  whose  route  goes 
through  i  will  need  to  change  state  anyway,  no 
matter  whether  there  is  another  stabilization 
wave  SWO'  other  than  5 WO  to  be  propagated 
by  i. 

In  this  case,  all  the  nodes  that  CWO  could 
have  reached  are  perturbed  nodes  and  need  to 
change  state.  Therefore,  the  existence  of  CW 0 
and  SW 0  does  not  introduce  any  contaminated 
nodes,  and  the  theorem  trivially  holds. 

—  If  i  does  not  need  to  change  state  in  the  pres¬ 
ence  of  faults  that  may  occur  to  nodes  in  Sp 
later,  then  action  CW  and  SW  will  be  enabled 
at  i  later  when  nodes  in  the  perturbed  region 
change  their  state.  The  latest  for  this  to  occur 
is  at  most  0(|SP|)  time  after  5 WO  is  initiated 
by  i,  since  a  containment  wave  must  be  initi¬ 
ated  in  Sp  and  it  takes  at  most  0(|5P|)  time 
for  the  containment  wave  to  reach  i  (note  that 
the  hop-length  of  the  longest  path  segment  in 
SP  is  Sp  ). 

When  action  CW  is  enabled  i,  a  contain¬ 
ment  wave  CWI  will  be  initiated/propagated 
which  propagates  along  the  same  paths  taken 
by  CWO  and  SWO.  Given  that  containment 
waves  propagate  faster  than  stabilization  waves 
do,  CWI  will  catch  up  with  and  stop  SWO,  at 
which  point  an  undo- containment  wave  UW  1 
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will  be  initiated  to  catch  up  with  CW 0.  When 
node  i  executes  action  SW,  i  initiates  an¬ 
other  stabilization  wave  SW  1  that  stabilizes 
the  state  of  nodes  which  have  been  affected  by 
5 WO  and  CW  1.  No  matter  when  SW  1  will 
be  initiated,  the  maximum  distance  CW 0  and 
5 WO  could  propagate  is  bounded  by  the  point 
where  UW 1  catches  up  with  CW 0. 

In  order  for  CW  1  to  be  able  to  catch  up  with 
and  stop  S' WO  in  the  above  process,  ds  > 
a-(dc  +  U)  should  hold,  given  that  the  propaga¬ 
tion  speeds  of  different  diffusing  waves  are  con¬ 
trolled  by  introducing  delays  in  action  execu¬ 
tion  when  a  node  schedules  its  enabled  actions, 
and  that  there  is  difference  between  clock  rates 
at  different  nodes.  Therefore,  the  farthest  dis¬ 
tance  dt  that  CWO  and  5 WO  could  propagate 
before  being  stopped  satisfies  the  following  for¬ 
mula:  0(|5P|)  +  (U  +  dc )  x  dt  x  +  (U  + 
du)  x  dt  x  (1  —  =  (L  +  dc)  x  dt ,  thus 

dt  ()(  Sp  ). 

Therefore,  the  theorem  holds  in  this  case. 

[Multiple  perturbed  regions]  When  there  are  mul¬ 
tiple  perturbed  regions,  we  need  to  prove  that  there  is 
no  interference  between  different  perturbed  regions.  To¬ 
ward  this  end,  we  first  define  the  contamination  region 
of  a  perturbed  region  5  as  the  set  of  nodes  that  are  con¬ 
taminated  by  S. 

Then,  if  the  the  contamination  regions  of  two  per¬ 
turbed  regions  are  disjoint,  the  claim  holds  trivially  since 
they  are  unable  to  affect  each  other  at  all.  Therefore,  we 
only  consider  the  case  where  the  contamination  regions 
of  two  or  more  perturbed  regions  are  adjoining  one  an¬ 
other. 

Without  loss  of  generality,  let  us  consider  two  per¬ 
turbed  regions  50  and  51,  whose  contamination  regions 
are  (750  and  (751  respectively,  and  (750  and  (751  are 
adjoining  each  other.  To  prove  that  50  and  51  do  not 
interfere  each  other,  we  only  need  to  prove  that,  without 
loss  of  generality,  the  containment  patterns  as  shown  in 
Figures  ??  and  ??  for  50  are  not  affected  by  any  node  kO 
that  is  contaminated  by  50  but  has  a  neighbor  kl  that 
is  contaminated  by  51.  This  is  the  case  because 

•  On  one  hand,  if  kO  does  not  change  state  because 
of  the  state  changes  at  kl,  then  the  existence  of  51 
does  not  affect  the  protocol  execution  within  the 
contamination  region  of  50,  and  thus  51  does  not 
affect  the  containment  activities  (by  the  execution 
of  action  CW  and  UW  in  the  contamination  region 
of  50)  for  50; 

•  On  the  other  hand,  if  kO  changes  state  because  of 
the  state  changes  at  kl  and  this  interrupts  the  exe¬ 
cution  of  containing  action  CW  or  UW  in  the  con¬ 
tamination  region  of  50,  this  must  be  due  to  the  fact 
that  the  corresponding  protocol  action  CW  or  UW 
becomes  disabled  at  kO  due  to  the  state  changes  at 


kl.  Therefore,  fcO  is  regarded  as  contaminated  by 
51,  and  kO  will  be  involved  in  the  containing  ac¬ 
tion  CW  or  UW  in  the  contamination  region  of  51. 
Thus,  the  state  changes  at  kO  will  be  contained  too 
even  in  this  case. 

Therefore,  the  theorem  holds  when  there  are  multiple 
perturbed  regions  at  a  state. 

□ 

For  cases  where  a  node  keeps  changing  its  state  (e.g., 
keeps  fail-stopping  and  rejoining  [5,  12,  19]),  we  have 
Theorem  2  (Stability-adaptive  control)  In  a  sys¬ 
tem  where  CPV  is  used,  the  distance  to  which  a  state 
Qi,k  of  a  node  i  propagates  is  6{ti,k),  where  U ik  is  the 
sojourn  time  of  state  qi 

Proof  :  To  prove  this  theorem,  let  us  consider  an  arbi¬ 
trary  pair  of  consecutive  states  and  qi,k+i  for  a  node 
i  where  the  sojourn  time  for  qiyk  is  U ik. 

When  i  changes  to  state  qi.k,  i  initiates  a  stabilization 
wave  SWik  (in  addition  to  a  containment  wave  CW^k-i 
that  is  to  contain  the  propagation  of  the  immediately 
previous  state  to  qijk  at  node  i).  When  i  changes  from 
state  qi:k  to  qi,k+ 1  after  tijk  time,  SWitk  would  have 
propagated  up  to  d‘^L  hops.  When  i  changes  to  state 
q^fc+i,  i  initiates  a  containment  wave  CWi }k  to  catch 
up  with  SWi.k-  The  farthest  distance  dt  from  a  node 
where  CWi }k  catches  up  with  5W* jk  to  i  satisfies  formula: 
dt  x  ( ds  +  L)  =  ti,k  +  dt  x  (dc  +  U)).  Therefore,  dt  = 
d  =  0(ti}k).  Similarly,  the  shortest  distance  dt' 

from  a  node  where  CWijk  catches  up  with  SWi.k  to  i 
satisfies  formula:  dt'  x  (ds  +  U)  =  ti:k  +  dt'  x  ( dc  +  L)). 
Therefore,  dt  =  de_d^u_L  =  6(ti}k). 

Thus,  the  theorem  holds. 

□ 

By  Theorem  2,  we  see  that,  in  CPV,  the  more  unsta¬ 
ble  a  node  is,  the  shorter  is  the  distance  to  which  its 
state  propagates  (since  the  sojourn  time  of  the  state  of 
a  more  unstable  node  is  shorter).  And  network  stabil¬ 
ity  is  improved  without  sacrificing  network  convergence 
in  the  sense  that  a  steady  state  reaches  outwards,  and  a 
transient  state  stays. 

For  cases  where  faults  stop  occurring  (either  indefi¬ 
nitely  or  for  a  long  enough  period)  after  a  point  in  time, 
we  have 

Theorem  3  (Local  stabilization)  Starting  at  an  ar¬ 
bitrary  state  qk  with  an  arbitrary  history  prefix  TL(qk)  and 
an  arbitrary  protocol  execution  £(qk),  the  system  com¬ 
putation  where  CPV  is  used  reaches  a  legitimate  state 
within  0{T(V(qk,/H(qk),£(qk))))  time  in  the  absence  of 
faults,  where  T  is  a  function  reflecting  the  route  ranking 
policy  used  in  the  system.  That  is,  a  system  where  CPV 
is  used  is  T -stabilizing. 

Proof:  Similar  to  the  proof  for  Theorem  1,  we  prove 
this  theorem  by  considering  the  two  different  cases  de¬ 
pending  on  whether  or  not  there  is  only  one  perturbed 
region  in  the  system. 
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[Single  perturbed  region]  When  there  is  only  one 
perturbed  region  Sp  in  the  network,  the  time  taken  for 
nodes  in  Sp  to  stabilize  is  0(r(|5p|))  according  to  the 
proof  for  Lemma  1  and  Theorem  1. 

By  Theorem  1,  the  contamination  range  of  Sp  is 
0(r(|S„|)).  Since  the  time  taken  for  the  nodes  in  the 
contamination  region  of  Sp  to  stabilize  if  proportional  to 
the  contamination  range  of  Sp,  the  time  taken  for  nodes 
contaminated  by  Sp  to  stabilize  is  0(T(|5P|)). 

When  the  time  taken  for  the  system  to  stabilize  is 
0(r(|S„|))  +  0(T(|5p|))  =  0(r(]5p|)).  When  Sp  is  the 
only  perturbed  region  in  the  network,  the  perturbed  size 
'P(Qk,'H(Qk),£(Qk))  =  |SP|.  Therefore,  the  system  sta¬ 
bilizes  within  0(T(V(qk,'H(qk),£(qk))))  time,  and  the 
claim  holds  in  this  case. 

[Multiple  perturbed  regions]  When  there  are  mul¬ 
tiple  perturbed  regions  S'. 0,  S.l,  . . . ,  and  S.m,  the  time 
taken  for  the  network  to  stabilize  depends  on  whether 
the  contamination  regions  of  these  perturbed  regions  are 
disjoint  or  not. 

When  the  contamination  regions  of  S.0,  S.l,  ..., 
and  S.m  are  disjoint,  the  stabilization  of  each  con¬ 
tamination  region  is  independent  of  that  of  the  other 
contamination  regions.  Therefore,  the  time  taken  for 
the  network  to  stabilize  is  O(maxj=o..m  r(jS.ij)).  Since 

v(qk,n(qk),£(qk))  =  E™ ol5-*l  and  r(E™olS'-*D  ^ 

maxj=o..m  r(|S.i|)  (by  the  definition  of  Timed  Dis¬ 
pute  Graphs  [?]),  the  network  stabilizes  within 
0{r{V{qk,‘H{qk),  £{qk))))  time. 

When  the  contamination  regions  of  two  perturbed  re¬ 
gions  S.k  and  S.(k  +  1)  are  adjoining,  the  stabiliza¬ 
tion  of  one  perturbed-region  may  depend  on  that  of 
the  other,  because  the  stabilization  in  path- vector  rout¬ 
ing  is  essentially  a  diffusing  computation  from  the  des¬ 
tination.  That  is,  one  perturbed  region  as  well  as 
its  contamination  region  can  stabilize  only  after  the 
other  perturbed  region  as  well  as  its  contamination 
region  has  stabilized.  Therefore,  in  the  worst  case 
where  the  set  of  perturbed  regions  as  well  as  their 
contamination  regions  stabilize  sequentially,  the  time 
taken  for  the  system  to  stabilize  is  the  summation  of 
the  stabilization  time  for  each  perturbed  regions,  i.e., 
°(Eilor(l‘S'-*l))-  Given  that  V{qk,'H{qk),£{qk))  = 
E^0IS'-*!  and  r(E^ols'il)  >  E^or(l5'-*!)  (by  the 

definition  of  Timed  Dispute  Graphs  and  the  fact  that 
T  represents  the  length  of  the  longest  simple  path  in  the 
Timed  Dispute  Graph  [?]),  the  network  stabilizes  within 

0(r(V(qk,'H(qk),£(qk))))  time. 

Therefore,  the  theorem  holds. 

□ 

In  the  common  case  where  the  SPF  policy  is  used, 
Theorem  3  implies 

Corollary  1  A  system  where  CPV  and  the  SPF  policy 
are  used  is  T -stabilizing,  and  T  is  a  linear  function. 

Proof:  From  the  proof  for  Theorem  3,  we  see  that  the 
time  taken  for  a  perturbed  region  Sp  as  well  as  its  con¬ 


tamination  region  to  stabilize  is  determined  by  the  time 
taken  for  Sp  to  stabilize,  which  is  0(r(|5p|)  in  general 
and  T  is  a  function  depending  on  the  route  ranking  pol¬ 
icy  used  in  the  system.  By  the  proof  for  Lemma  1,  we 
see  that,  when  the  SPF  policy  is  used,  the  time  taken  for 
a  perturbed  region  Sp  to  stabilize  is  0(j5pj).  Therefore, 
this  corollary  trivially  holds. 

□ 

By  the  analysis  above,  we  see  that  CPV  contains  con¬ 
tinuously  occurring  faults  and  locally  stabilizes;  the  de¬ 
gree  of  fault  containment  and  the  time  taken  to  stabilize 
is  a  function  of  the  perturbation  size  instead  of  the  net¬ 
work  size.  We  also  see  that  CPV  is  “stability-adaptive” 
in  the  sense  that  the  state  of  stable  nodes  propagate 
outwards,  and  the  state  of  unstable  nodes  is  locally  con¬ 
tained. 

In  the  next  section,  we  corroborate  our  analysis  by 
simulating  Internet-like  networks  and  studying  how  con¬ 
tinuous  containment  and  local  stabilization  improve 
packet-forwarding  in  the  presence  of  continuously  occur¬ 
ring  faults. 

6  Simulation  results 

We  have  implemented  CPV  in  SSFNet  [1],  a  network 
simulator  which  supports  a  rich  set  of  standard  Inter¬ 
net  protocols  such  as  BGP  (with  route- flap-damping). 
In  Section  5,  we  have  analyzed  continuous  containment 
and  local  stabilization  in  CPV;  in  this  section,  therefore, 
we  focus  on  the  impact  of  continuous  containment  and 
local  stabilization  on  packet  forwarding  in  the  presence 
of  continuously  occurring  faults. 

In  our  simulation,  CPV  is  executed  without  using 
any  existing  instability-suppression  mechanisms;  BGP  is 
executed  with  instability-suppression  timers  and  route- 
flap-damping  with  the  standard  parameter  setup.  For 
comparability,  we  set  parameter  ds  of  CPV  as  30  sec¬ 
onds,  which  is  the  default  MinRouteAdvertisementlnter- 
val  value  used  in  BGP.  Accordingly,  we  set  dc  and  du  as 
10  and  1  seconds  respectively  for  CPV. 

For  fidelity  of  simulation,  we  use  realistic  Internet-type 
topologies  [1]  to  evaluate  CPV.  And  to  study  the  impact 
of  network  size,  we  use  networks  of  size  ranging  from 

7  ASes  to  75  ASes.  To  simulate  continuously  occurring 
faults,  we  let  an  arbitrary  node  j  repeatedly  fail-stop  and 
then  rejoin  every  30  seconds.  The  simulation  results  are 
as  follows. 

Continuous  containment.  Figure  4  shows  the  maxi¬ 
mum  contamination  range  and  the  maximum  number  of 
nodes  affected  (i.e.,  perturbed  or  contaminated)  by  the 
faults  in  BGP  and  CPV.  We  see  that  the  contamination 
range  and  the  number  of  nodes  affected  by  the  faults  in 
BGP  increase  as  network  size  increases,  and  every  node 
is  affected  by  the  faults  in  BGP;  whereas  in  CPV,  the 
contamination  range  remains  1  and  the  number  of  nodes 
affected  by  the  faults  remains  small  as  network  size  in- 
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Figure  6:  Impact  of  continuously  occurring  faults 


creases,  since  CPV  contains  the  impact  of  the  continu¬ 
ously  occurring  faults  locally  around  where  they  occur 
(as  also  proved  in  Theorem  1). 

Local  stabilization.  Figure  5  shows  the  time  taken 
for  BGP  and  CPV  to  stabilize  once  faults  stop  occurring 
when  j  is  up  (we  discuss  the  case  where  a  node  fail-stops 
in  Section  7).  We  see  that  the  time  taken  for  BGP  to 
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Figure  7:  Time  taken  to  stabilize 

stabilize  increases  as  network  size  increases;  whereas  the 
time  taken  for  CPV  to  stabilize  remains  small  as  network 
size  increases,  since  the  time  taken  for  CPV  to  stabilize 
depends  on  the  perturbation  size  instead  of  the  network 


size  (as  also  proved  in  Theorem  3),  and  the  perturbation 
size  remains  small  in  CPV  as  network  size  increases. 
Stability-adaptive  control.  To  study  the  property  of 
stability-adaptiveness  in  CPV,  we  let  an  arbitrary  node 
oscillate  among  different  states  and  study  the  relation¬ 
ship  between  the  sojourn  time  of  a  state  and  the  distance 
to  which  the  state  propagates  (i.e.,  the  maximum  dis¬ 
tance  from  the  oscillating  node  to  the  nodes  that  adapt 
to  the  state).  The  results  are  shown  in  Figure  6.  We 
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Figure  8:  Stability-adaptiveness  of  CPV 

see  that  states  that  last  longer  propagate  farther  and, 
accordingly,  more  nodes  adapt  behaviors;  as  the  sojourn 
time  of  a  state  becomes  shorter  and  shorter,  the  distance 
to  which  the  state  propagates  decreases  more  and  more 
(as  also  proved  in  Theorem  2). 

7  Discussion 

In  this  section,  we  discuss  approaches  to  further  improve 
the  performance  of  CPV,  mechanisms  to  implement  CPV 
efficiently,  and  incremental  deployment  of  CPV. 
Sub-linear  containment  &:  stabilization.  We 
mainly  focused  on  the  containment  of  continuously  oc¬ 
curring  faults  in  this  paper,  and  the  contamination  range 
as  well  as  the  convergence  time  (after  faults  stop  occur¬ 
ring)  is  a  linear  function  of  the  perturbation  size  in  the 
common  case  where  the  SPF  policy  is  used.  If  we  apply, 
together  with  CPV,  mechanisms  that  expedite  the  con¬ 
vergence  of  path- vector  routing  protocols  (such  as  those 
proposed  in  [13]  and  [20]),  then  the  contamination  range 
of  a  perturbed-region  is  reduced  to  a  linear  function  of 
the  diameter  of  the  region,  and  thus  both  the  contami¬ 
nation  range  and  the  convergence  time  are  reduced  to  a 
sub- linear  function  of  the  perturbation  size. 
Implementation  of  CPV.  In  CPV,  neighboring  nodes 
exchange  their  state  periodically,  which  is  required  for 
protocols  to  stabilize  from  state  corruptions.  To  reduce 
the  overhead  of  the  periodic  information  synchroniza¬ 
tion,  we  can  apply  the  technique  proposed  in  [18]  that 
guarantees  the  consistency  of  the  routing-tables  between 
neighboring  nodes  in  a  scalable  manner. 

In  CPV,  when  network  state  changes,  a  node  i  main¬ 
tains  variable  i.tp  to  denote  the  next  route  that  i  will 
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adopt.  Since  i.tp  is  transient  and  is  meaningful  only 
during  the  stabilization  of  CPV,  each  node  i  only  needs 
to  maintain  i.tp  for  destinations  to  which  the  route  of  i 
needs  to  change.  Therefore,  the  use  of  i.tp  in  CPV  does 
not  introduce  much  memory  overhead,  since  the  major¬ 
ity  of  the  network  is  expected  to  be  stable  most  of  the 
time. 

Incremental  deployment  of  CPV.  Containment 
waves  in  CPV  introduce  new  information  other  than  that 
used  in  BGP.  To  enable  graceful  migration  of  and  inter¬ 
operability  with  BGP,  we  define  a  new  optional  transi¬ 
tive  path  attribute  [16],  and  use  this  attribute  to  encode 
the  information  newly  introduced  in  CPV.  According  to 
BGP  specification,  a  BGP  speaker  propagates  a  tran¬ 
sitive  attribute  upon  receipt,  whether  or  not  the  BGP 
speaker  implements  CPV.  Therefore,  CPV  can  be  in¬ 
crementally  deployed  and  inter-operate  well  with  BGP. 
Moreover,  even  in  the  case  of  partial  deployment,  CPV 
can  help  contain  faults  that  keep  occurring  to  a  region 
where  CPV  is  used. 

8  Concluding  remarks 

To  characterize  system  properties  in  the  presence  of 
continuously  occurring  faults,  we  formulated  the  no¬ 
tions  of  perturbed  node,  contaminated  node,  perturba¬ 
tion  size,  contamination  range,  JP-containment,  and  T- 
stabilization.  These  concepts  are  generically  applicable 
to  networking  and  distributed  computing  problems.  In 
general,  a  self-stabilizing  protocol  that  contains  continu¬ 
ously  occurring  faults  also  locally  stabilizes,  and  a  locally 
stabilizing  protocol  contains  faults  that  occur  only  once 
and  at  the  same  time. 

We  designed  the  path-vector  routing  protocol  CPV 
that  contains  continuously  occurring  faults  and  locally 
stabilizes.  In  CPV,  the  distance  to  which  the  state  of 
a  node  propagates  is  proportional  to  the  sojourn  time 
of  the  state.  CPV  achieves  these  properties  by  lay¬ 
ering  a  system  computation  into  three  diffusing  waves 
(i.e.,  stabilization  wave,  containment  wave,  and  undo- 
containment  wave)  which  run  in  parallel  and  coordinate 
to  contain  the  propagation  of  obsolete  information  while 
stabilizing  a  network  at  the  same  time.  Built  upon  mod¬ 
els  for  the  Internet,  CPV  is  readily  applicable. 

In  this  paper,  we  focused  on  how  to  contain  high- 
frequency  unanticipated  faults  without  detecting  why 
and  where  the  faults  occur.  This  design  enables  CPV 
to  work  both  when  faults  are  benign  and  transient  in¬ 
stability  is  unavoidable  [20]  as  well  as  in  adversarial 
cases  where  nodes  are  compromised  [12,  19].  In  the  lat¬ 
ter  cases,  CPV  can  be  applied  together  with  security 
and  fault  diagnostic  mechanisms  that  detect  and  sup¬ 
press  mis-behaving  nodes  or  links;  since  CPV  contains 
faults  locally  around  where  they  occur,  the  security  and 
fault  diagnostic  mechanisms  can  be  more  optimistic  (e.g., 
higher  cutoff  threshold  and  shorter  suppression  time  in 


route  flap  damping  [17])  to  avoid  issues  such  as  severely 
delayed  convergence  in  BGP  after  route  flap  damping. 
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